Frederick J. Kelly invented multiple-choice question (MCQ) format in 1914, primarily to democratize education and wring out bias from student evaluation. Hence the alternative monikers “Objective,” or “standardized” have been used for MCQs. While the colleges of Kelley's day were slow to adopt it, multiple-choice is now the world's most commonly employed testing format. Its use has grown even more rapidly since the advent of computerized tests, especially for examinations involving large numbers of students, presumably because MCQs lend themselves readily to automation.
In the last one hundred years multiple-choice format has essentially remained the same, despite its ubiquity and widely recognized shortcomings. A key reason for its popularity, it is safe to say, is this possibility of automation, from which flow the other advantages, such as, objectivity, uniformity and scalability of testing and grading. Open format, long or short essays are no match for MCQ format, as they are notoriously subjective, non-uniform and non-scalable, hence expensive to administer. The Reverse Multiple-choice Method was devised by one of the present inventors to address some of the shortcomings of multiple-choice testing format.
Multiple-choice questions remain a common way of testing students in virtually all subject areas, particularly in examinations taken by large numbers of students. In its most commonly used form, a multiple-choice question comprises three identifiable sections: a section containing a set of facts to be presumed, an interrogative sentence, and a set of answer choices. Together, the first and the second section, may be referenced as a “query.” A student generally answers an MCQ by indicating the answer selection from the given answer choices. Whether the test is conducted on a computer or paper, grading can be automated or carried out nearly automatically.
By now, however, the long experience with multiple-choice tests in many academic settings has shown that these advantages of MCQs can turn into its disadvantages: The very ease and simplicity of selecting and recording an answer to a test question blurs the difference between knowledge and ignorance. All that the examinee has to do is click or mark an answer, which is possible to do completely at random with nary an inkling of the subject matter. The probability of scoring a passing grade is very low if an examinee is completely ignorant, but it is possible to improve the scores with a little knowledge and answer guessing techniques. Unlike the open format/essay testing format, a test-taker generally has little scope to actively reveal knowledge in a multiple-choice test, therefore cheating is harder for the examiner to spot when it occurs.
Reverse Multiple-Choice Method (“RMCM”) was introduced in U.S. patent application Ser. No. 09/951,132, now U.S. Pat. No. 7,033,182, and extended in patent application Ser. No. 11/266,350, now U.S. Pat. No. 8,195,085. RMCM offers a promising alternative for teaching, training and assessment that may be automated/computerized, and thereby inherits many advantages of multiple-choice format. RMCM technology combines the uniformity, efficiency and grading ease of “objective” or “standardized” multiple-choice tests with the reliable “measure of knowledge and understanding” generally associated with open/long format tests.
The present invention stems from the observation that the advantages of RMCM are not limited to interactions between human teachers/examiners and students/examinees, but that it is possible to adapt and extend RMCM to the case where at least one of the communicants in an interaction, e.g., between teacher/trainer and student/trainee, is a machine. Such extension of RMCM, however, must respect the differences between humans and computerized machines as to the learning capacity and the accumulation, retention, recall of knowledge, as well as the “tacit knowledge” of the application environment. This invention relates to the improvement of RMCM when at least one party to such communication is a machine.